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I.  STATEMENT  OF  RESEARCH  PROBLEM 


Much  research  and  development  effort  has  been  devoted  to 
create  smart  weapon  systems  capable  of  detecting  air  and  ground 
targets  in  widely  varying  hostile  environments.  It  is  generally 
accepted  that  currently  available  multispectral  sensor  technology 
(infrared,  visible,  ladar,  millimeter  wave)  provides  sufficient 
information  to  develop  robust  vision  systems  that  perform  reliably 
in  complex  scenes  with  widely  varying  environmental  and 
countermeasure  conditions.  This  acceptance,  however,  is  often 
based  upon  the  ability  of  a  human  observer  to  perform  the  visual 
tasks  using  the  sensory  data.  The  basic  problem  is  to  select 
multiple  sensors /features  that  provide  sufficient  information  to 
allow  the  development  of  robust  algorithms  that  can  resolve  the 
complexities  introduced  by  dynamic  environmental  and  countermeasure 
conditions.  Robust  multiple  sensor/ feature  processing  algorithms 
to  distinguish  target  signatures,  however,  have  not  demonstrated 
the  ability  to  reliably  distinguish  signatures  in  widely  varying 
hostile  environments.  The  result  has  been  the  development  of  a 
large  number  of  algorithms  that  work  only  in  very  limited 
conditions.  Much  effort  is  spent  trying  to  modify  these  algorithms 
to  work  in  a  larger  classes  of  environments.  These  efforts  have 
generally  been  unsuccessful  due  to  the  fact  that  there  is  no 
validated  theory  for  adaptively  selecting  and  fusing  the 
distinguishing  features  in  terms  of  the  large  number  of 
environmental  and  countermeasure  attributes  that  exist  in  real 
battlefield  environments.  Furthermore,  these  algorithms  do  not 
utilize  the  important  physical  principles  of  the  atmosphere  and 
countermeasures  to  distinguish  target  end  background  signatures. 

The  research  sponsored  by  this  grant  is  focused  on  two 
important  technical  barriers  that  inhibit  the  development  of  robust 
vision  systems.  These  are: 

*  Robust  scene  analysis  algorithms  to  reliably  analyze 
complex  scenes  with  widely  varying  environmental  and 
countermeasure  conditions. 

*  Real  time  processing  structures  to  assimilate  large 
quantities  of  multisensor  information  to  make  optimum 
decisions. 

The  primary  objective  of  this  research  grant  is  to  develop  a 
theoretical  and  technological  basis  for  overcoming  these  barriers. 
Our  approach  to  these  barriers  is  to  develop  mathematical  methods 
to  evaluate  and  fuse  multisensor  features  to  analyze  complex  scenes 
using  hybrid  optical/digital  processing  techniques.  Rapid 
evaluation  of  individual  and  multiple  sensor  performance  together 
with  effective  sensor  fusion  are  essential  capabilities. 
Intelligent  vision  systems  must  quickly  analyze  dynamic  visual 
scenes  and  produce  intricate  non-linear  control  strategies.  These 
tasks  are  complicated  by  the  fact  that  the  sensory  information  is 
often  uncertain,  incomplete  and  even  contradictory. 
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II.  SUMMARY  OF  RESULTS 


The  research  performed  during  this  grant  is  focused  on  the 
development  of  mathematical  foundations  for  designing  robust 
multisensor  vision  systems.  Emphasis  is  placed  on  four  major 
research  areas.  These  are: 

*  Electronic  Vision/ Image  Processing 

*  Electro-Optical  Processing 

*  Vision  Systems/ Intelligent  Control 

*  Multiprocessor  Vision  Architecture 

The  first  area  concerns  image  processing  functions  for  fusing, 
detecting  and  recognizing  objects  of  interest  in  multisensor  image 
sequences.  The  second  research  area  concerns  the  advancement  of 
electro-optical  processing  techniques  and  integration  with  digital 
processing  techniques  to  detect  and  recognize  objects.  The  third 
area  involves  the  development  and  testing  of  a  control  model  to 
support  the  development  of  intelligent  control  algorithms..  The 
last  area  involves  the  development  of  a  methodology  for  designing 
and  evaluating  multiprocessor  architectures  for  real  time  vision 
systems . 

2.1  Image  Processing  Research 

The  thrust  of  the  image  processing  research  is  to  develop 
mathematical/engineering  foundations  for  designing  and  evaluating 
multisensor  vision  systems  capable  of  performing  in  widely  varying 
and  hostile  environments.  The  major  accomplishments  of  this 
research  effort  are: 

*  Theoretical  foundations  and  algorithms  were  developed  for 
evaluating  the  complexity  [1]  of  vision  tasks  in  terms  of  the 
information  content  of  multisensory  inputs.  This  allows 
sensors  and  features  to  be  designed  to  reduce  the  complexity 
to  meet  design  specifications. 

*  Statistical  measures  [2-5]  were  established  for  evaluating  the 
performance  of  multisensor  features  in  terms  of  their  ability 
to  distinguish  target  and  background  signatures.  These 
measures  are  effective  indicators  of  performance  in  high¬ 
dimensional  sensor/ feature  spaces. 

*  A  laboratory  environment  [31]  was  developed  for  integrating 
optical  and  digital  processing  concepts.  Optical  and  digital 
image  processors  have  been  successfully  integrated,  resulting 
in  a  high  speed  hybrid  optical/digital  scene  analysis  system 
that  combines  the  speed  and  massive  parallelism  of  optical 
processors  with  the  flexibility  and  statistical  decision¬ 
making  capability  of  digital  processors.  The  vision  system 
utilizes  a  large  f ield-of-view  camera  to  locate  objects  of 
interest  and  direct  a  high  resolution  color  camera  to  segment 
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and  recognize  objects  in  a  cluttered  background  scene. 
Optical  processing  techniques  perforin  the  intensive 
computational  tasks  associated  with  locating  and  recognizing 
objects  of  interest.  Currently,  the  prototype  system  can 
recognize  any  one  of  nine  similar  objects  placed  at  any 
position  and  orientation  on  the  NMSU  terrain  board. 

*  An  adaptive  "neural-like"  decision  module  [10]  was  developed 
for  making  fast  and  near  minimal  probability  of  error 
decisions  in  high  dimensional  sensor/ feature  spaces.  Using 
this  decision  module,  robust  algorithms  were  developed  for 
target  detection,  segmentation  and  identification.  These 
algorithms  are  incorporated  in  our  optical/digital  electronic 
vision  system  and  have  demonstrated  robust  performance  in 
complex  scenes. 

*  Significant  progress  has  been  achieved  in  developing  a  common 
mathematical  foundation  for  the  comparative  analysis  of 
statistical,  fuzzy  logic,  and  artificial  neural  network 
pattern  recognition  systems  [12].  Practical  applications  of 
this  theory  can  combine  previously  disjoint  disciplines  into 
hybrid  pattern  recognition  systems  that  can  rapidly  learn  and 
adapt  to  changing  environments. 

*  Researchers  at  NMSU  and  the  Atmospheric  Sciences  Laboratory 
(ASL)  are  jointly  investigating  the  effects  of  atmospheric  and 
field  conditions  on  sensor/ feature  performance  [13,14]. 
Atmospheric  turbulence  phenomena  in  the  long  infrared  have 
been  isolated  in  data  obtained  by  the  ASL  Target  Contrast 
Character izer.  Significant  progress  has  been  made  in 
developing  and  validating  turbulence  models  with  the  field 
data.  The  goal  of  this  cooperative  effort  is  to  develop  new 
techniques  for  processing  infrared  imagery  degraded  by 
atmospheric  turbulence  and  ultimately  to  develop  new  systems 
with  significantly  improved  performance. 

2 1.L-A _ Complexity  of  vision  Tasks 

A  fundamental  problem  in  the  design  of  vision  systems  is  to 
assess  the  level  of  difficulty  associated  with  performing  the 
required  tasks  with  the  sensor  information  available.  A  doctoral 
dissertation  [1]  was  associated  with  developing  effective  measures 
to  establish  task  complexity  in  terms  of  sensor/ features.  These 
complexity  measures  were  developed  in  terms  of  a  random  process 
model  of  the  scene  environment  as  viewed  by  the  sensor  array.  The 
complexity  measures  are  all  developed  in  terms  of  optimal 
performance  levels  for  statistical  decision  rules,  and  therefor 
provide  achievable  bounds  on  system  performance.  The  measures  can 
be  used  to  evaluate  and  select  features  and  algorithms  based  on 
their  effectiveness  in  the  decision-making  process.  Theorems  and 
proofs  establish  important  and  useful  properties  of  the  measures 
and  relate  them  to  other  known  performance  measures.  These 
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measures  have  been  widely  used  to  evaluate  the  effect  of 
countermeasures  and  atmospheric  disturbances  on  sensor/ feature 
performance. 

2.1.2  Information  Fusion 

Two  doctoral  dissertations  [2,3]  were  associated  with 
developing  a  mathematical  basis  for  evaluating  and  fusing  sensory 
information  for  optimal  decision  processes.  Effective  methods  were 
established  for  evaluating  the  performance  of  multisensor  features 
in  terms  of  their  ability  to  distinguish  target  and  background 
signatures.  A  statistic,  called  the  tie  statistic,  was  introduced 
[2]  and  developed  as  a  measure  for  distinguishing  two  probability 
density  functions.  Theorems  and  proofs  establish  important 
relations  between  the  tie  statistic  and  the  Kolmogorov-Smirnov 
measures.  Through  these  relations,  the  tie  statistic  is  related  to 
other  popular  discriminating  measures  such  as  the  Matusita 
distance,  the  Bhattacharyya  coefficient  and  the  divergence  measure. 
The  tie  statistic  can  be  used  to  transform  a  feature  space  to  an 
ordered  space  with  simplified  decision  boundaries.  The  mapping 
process  uses  the  tie  statistic  to  measure  differences  between 
probability  density  functions  of  features.  Hence,  the  new  tie 
space  can  be  used  to  evaluate  the  difficulty  associated  with 
distinguishing  signatures  in  high  dimensional  feature  spaces. 
Necessary  and  sufficient  condition  were  established  [3,4,5]  for 
using  the  tie  statistic  to  fuse  multisensor  date  into  a  single 
signal  without  increasing  the  complexity  of  the  vision  task. 
Algorithms  were  developed  [4,6]  to  perform  information  fusion, 
texture  recognition  and  target  cueing  in  multisensor  scenes. 

2:1,3 _ K-Nearest  Neighbor  signature  Analysis 

The  k-nearest  neighbor  concept  [7]  was  used  to  develop  a 
measure  for  evaluating  the  difficulty  associated  with 
distinguishing  signatures  in  high  dimensional  feature  spaces.  The 
new  measure,  called  the  k-complexity,  combines  the  complexity 
measures  [1,2]  and  the  k-nearest  neighborhood  concept  to  provide  an 
effective  method  to  estimate  the  minimum  probability  of  error 
associated  with  distinguishing  target  signatures.  The  major 
advantage  of  the  k-complexity  is  that  it  can  be  computed  without 
estimating  the  joint  probability  density  functions.  This  is 
particularly  important  for  multispectral  signature  analysis  where 
the  number  of  sensors/ features  of  interest  can  be  quite  large. 
Efficient  software  modules  have  been  developed  to  compute  the  k- 
complexity  and  experiments  have  demonstrated  its  potential. 

The  k-complexity  measure  is  defined  in  terms  of  the  k-nearest 
neighborhood  concept  often  used  for  estimating  probability  density 
functions.  Given  two  sets  of  signature  measurements 

T  *  {  X  :  X  =  (  xl,x2,...,xN  )  target  feature  vector  } 

B  *  {  X  :  X  *  (  xl,x2,...,xN  )  background  feature  vector  } 
each  with  NS  samples  that  characterize  the  target  and  background 
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signatures,  respectively.  Considering  the  T  and  B  samples  as 
points  in  an  N  dimensional  space,  the  k-nearest  neighborhood  set, 
KNN(X),  is  defined  as  the  k-nearest  points  to  X  in  the  set  T  u  B. 
A  discrete  random  variable  Y  is  defined  as  the  number  of  points  in 
the  intersection  of  KNN(X)  and  T,  i.e., 

Y  =  order (KNN (X)  n  T)  Ye  {0,1,..., K}. 

The  conditional  probability  density  functions  f(y:XeT)  and 
f(y:XeB)  are  defined  as  the  probability  of  observing  y  given  that 
X  came  from  T  or  B,  respectively.  Letting  Pt  and  Pb  define  the  a 
priori  probabilities  of  target  and  backgrounds  observations,  the  k- 
complexity  is  defined  by 

i  K 

KC=  (PtApb)  E  Ptf(y-XeT)KPbf(y:XeB)  . 


where  the  operator  A  selects  the  minimum.  It  is  important  to 
observe  that  the  K-corplexity  is  defined  in  terms  of  single 
dimensional  pdf's,  making  it  computationally  attractive  for 
estimating  the  effectiveness  of  multi-dimensional  feature  vectors. 
The  k-complexity  varies  from  0  to  1  and  measures  the  level  of 
difficulty  associated  with  detecting  whether  the  measurements  came 
from  a  target  or  from  the  background.  If  the  k-complexity  is  a 
zero  then  there  is  little  or  no  chance  of  confusing  target  and 
background  measurements.  On  the  other  hand,  if  k-complexity  is  one 
then  there  is  an  even  chance  based  on  the  measured  features  that 
the  measurements  came  from  a  target  or  background. 

Several  important  theorems  [8]  have  been  established  to  give 
the  k-complexity  measure  credibility.  If  the  likelihood  ratio  of 
the  target  to  background  is  a  constant  in  the  hyper-spheres  defined 
by  KNN(X) ,  then  the  k-complexity  is  directly  related  to  the  minimum 
probability  of  error  (MPE)  given  by  MPE  =  0.5*KC.  Furthermore,  in 
constant  likelihood  ratio  case,  the  k-complexity  approaches  the 
complexity  defined  in  terms  of  the  joint  probability  density 
functions  and  is  independent  of  k.  Experimental  results  [9]  have 
shown  that  good  estimates  of  k-complexity  are  obtained  with  very 
reasonable  sample  sizes.  The  direct  relation  between  the  k- 
complexity  and  MPE  along  with  the  fact  that  good  estimates  can  be 
obtained  with  relatively  small  samples,  motivated  the  development 
of  the  k-complexity  to  form  a  basis  for  measuring  the  effectiveness 
of  multispectral  feature  vectors  in  distinguishing  target 
signatures  in  CM  environments. 

2.1.4  Adaptive  Decision  Module 

An  adaptive  "neural-like"  decision  module  [10]  was  developed 
for  making  fast  and  near  minimal  probability  of  error  decisions  in 
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high  dimensional  sensor/ feature  spaces.  New  algorithms  for 
clustering  high  dimensional  multisensor  data  and  modeling  the 
clusters  with  extended  k-nearest  neighbor  concepts  have  been 
developed.  The  theory  of  k-nearest  neighbor  concept  has  been 
extended  to  generate  quadratic  surfaces  by  weighting  the  points. 
Any  quadratic  surface  can  be  realized  with  only  two  control  points. 
Adaptive  algorithms  have  been  developed  to  locate  the  control 
points  and  weights  to  realize  multisensor  decision  surfaces.  Using 
these  concepts,  several  new  algorithms  for  target  detection  and 
identification  have  been  incorporated  in  a  demonstration 
optical/digital  electronic  vision  system.  The  new  algorithms  have 
demonstrated  robust  performance  in  cluttered  scenes. 

The  adaptive  decision  module  [10]  has  been  used  to  perform 
complex  and  nonlinear  control  functions.  A  training  procedure  is 
used  to  generate  a  finite  set  of  control  points  in  the  state  space 
that  define  the  control  law.  The  controller  has  been  used  to 
balance  an  inverted  pendulum  mounted  on  a  moveable  cart  and  to 
remotely  position  a  trailer  truck  to  a  specified  position  in  a 
constrained  region  using  a  video  tracking  system.  These  problems 
demonstrate  the  power  and  simplicity  of  the  nearest  neighbor 
controller  in  nonlinear  systems.  To  demonstrate  the  concepts,  a 
vision  guided  system  was  developed  to  remotely  park  a  truck-trailer 
in  a  constrained  region.  The  control  structure  adapts  well  to 
random  effects  caused  by  measurement  and  vehicle  control  errors  and 
is  suited  for  nonlinear  and  unstructured  control  problems  that  are 
difficult  to  model. 

_ Mathematical  Foundations  for  Pattern  Recognition  Systems 

A  common  mathematical  foundation  for  the  comparative  analysis 
of  statistical,  fuzzy,  and  artificial  neural  pattern  recognition  or 
decision  making  systems  was  developed  [12]  using  abstract  algebraic 
techniques  to  characterize  the  functions  generating  decision 
surfaces  and  the  learning/training  processes  involved  in  each 
technique . 

Abstract  algebra  is  built  on  the  most  basic  of  foundations: 
sets,  operations  on  sets,  and  mappings  from  set  to  set.  Viewed 
from  this  level,  all  pattern  recognition  methods  are  strikingly 
similar.  So  much  so,  that  a  generalized  pattern  recognizer  (GPR) 
can  be  defined  such  that  all  others  are  its  subsets  or  special 
cases.  The  basic  structure  of  the  GPR  was  developed  during  this 
project.  Typical  popular  statistical,  fuzzy,  and  artificial  neural 
pattern  recognizers  were  characterized  in  the  context  of  the  GPR 
[12].  Each  of  the  modern  methods  has  strengths  and  weaknesses, 
advantages  and  disadvantages.  Using  the  concept  of  a  GPR,  it  is 
possible  to  design  and  build  hybrid  systems  that  capitalize  on  the 
advantages  and  strengths  of  all  previous  methodologies.  Each  of 
the  modern  pattern  recognition  areas  has  its  own  terminology  and 
set  of  notations.  As  much  as  possible.,  the  standard  notation  for 
each  discipline  was  preserved  and  described  in  its  relation  to  the 
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standard  notation  of  abstract  algebra. 

The  study  of  pattern  recognition  has  many  important  goals 
among  them  are  the  development  of  methodologies  for  designing  and 
building  machines  capable  of  recognizing  and  learning  to  recognize 
objects  in  natural  or  unstructured  environments.  The  common  task 
of  any  pattern  recognizer  is  to  obtain  a  set  of  observations  or 
measurements  from  an  unknown  (or  unclassified)  pattern  and  to 
decide  which  class  the  pattern  belongs  based  on  the  values 
observed.  From  an  abstract  point  of  view,  once  pattern  recognition 
is  clearly  defined  within  a  mathematical  framework,  the  pattern 
recognition  task  becomes  simply  a  mathematical  problem. 

A  pattern  recognizer  is  a 
decision  maker  or  decider,  D, 
that  can  be  viewed  as  a  mapping 

from  an  arbitrary  observation  Abstract  pattern  recognizer 

space  M  to  a  finite  set  of 
symbols  representing  decisions 
or  object  classes,  ft.  D  is 
therefore  a  triple  ( M,Cl,6 )  with 
M,  a  space  (a  largest  set  from 
which  subsets  of  observations 
are  taken)  ,  ft,  a  finite  set  of 
symbols,  and  6,  a  mapping  from 
M  to  n,  The  decider  is 

meaningful  only  if  n  contains 
two  or  more  elements  and  M  Fig.  i  conceptual  view  of 

contains  at  least  as  many  abstract  pattern  recognizer, 

elements  as  ft.  For  most 

deciders,  the  mapping  <5  is  many- 

to-one  with  the  number  of  elements  in  M  quite  large,  often  not 
finite.  In  these  cases,  the  mechanism  creating  6  defines  a 

partition  of  M  with  non-overlapping  subsets  of  M  mapping  into 

elements  of  ft.  Note  that  members  of  the  special  subset  of  deciders 
characterized  by  finite  M  are  called  combinatoric  machines  in 
automata  theory  with  M  being  a  finite  set  of  input  symbols  and  ft  a 
finite  set  of  output  symbols  [10]. 

There  are  three  basic  types  of  deciders:  fixed,  trainable,  and 
learning.  The  mapping  <5  in  a  fixed  decider  is  static.  It  is 
developed  prior  to  the  use  of  the  decider  and  does  not  change 
during  decider  operation.  The  trainable  decider  has  a  training 
mode  wherein  the  mapping  can  be  altered  in  response  one  or  more 
sets  of  measurements  whose  classification  is  known  a  priori.  The 
mapping  of  a  learning  decider  develops  and  evolves  as  the  decider 
encounters  and  classifies  new  patterns  of  unknown  classes.  There 
are,  of  course,  many  variations  of  these  basic  types. 

Irregardless  of  the  type  or  implementation  method,  all  pattern 
recognizers  are  deciders  as  defined  above.  Once  a  specific  pattern 
recognition  task  has  been  defined,  the  sets  AT  and  ft  are  the  same 
whether  the  pattern  recognizer  be  classical  deterministic. 
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statistical,  fuzzy,  artificial  neural,  or  something  altogether  new. 
What  differs  for  each  is  the  mapping  S  and  its  generation.  For 
typical  examples  of  each  of  the  pattern  recognition  methods  chosen 
for  comparative  analysis,  the  composition  of  S  is  derived  and  used 
as  a  basis  for  comparison  and  contrast.  In  each  of  these  methods, 
the  mapping  S  is  expressed  as  a  composition  of  a  decision  rule  and 
another  set  of  mappings  classically  termed  discriminant  or  decision 
functions.  The  three  methods  chosen  for  analysis  are  also  based  on 
the  implicit  assumption  that  M  and  the  subsets  of  M  from  which 
measurements  are  taken  is  a  measurable  space.  In  this  development, 
M  was  arbitrarily  restricted  to  be  a  vector  space. 

These  restrictions  were  for  convenience  only  and  are  not 
contained  in  the  fundamental  definition  of  a  decider.  Without 
these  restrictions,  extraordinarily  rich  pattern  recognizers  can  be 
developed  using  non-measurable  spaces  containing  measurable 
subspaces.  The  use  of  a  non-measurable  space  M  provides  a 
mechanism  for  incorporating  non-maasurable  information  dimensions 
and  non-measurable  a  priori  knowledge  into  a  pattern  recognizer. 
This  important  extension  was  beyond  the  scope  of  the  present  work 
and  was  left  for  future  research. 

2.1.6  Effects  of  Atmospheric  Phenomena  on  Sensor  Performance 

Modern  high  resolution  infrared  (IR)  sensors  are  instrumental 
in  advanced  systems  for  automatic  detection  and  recognition  of 
interesting  objects  in  cluttered  backgrounds  along  relatively  long 
(l-2km)  horizontal  paths.  The  thermal  sensing  component  of  a 
Automatic  Target  Recognizer  (ATR)  is  called  upon  to  assist  in 
locating  regions  containing  objects  or  targets  of  interest, 
separating  the  objects  from  the  background,  classifying  objects  and 
background  scenes,  and  making  control  decisions.  These  tasks  are 
commonly  referred  to  as  the  cuer,  segmenter,  recognizer,  and 
controller  respectively.  The  design  of  these  processes  begins  by 
selecting  a  set  of  sensors  and  features  that  can  distinguish  object 
versus  background  signatures.  Typically,  the  selection  of  sensors 
and  features  is  based  on  models  of  physical  properties  of  the 
objects  and  backgrounds  of  interest.  The  goal  is  to  remotely  sense 
and  extract  sufficient  information  to  distinguish  object  and 
background  signatures  in  widely  varying  environmental  conditions. 
Algorithms  based  on  statistical,  syntactical,  or  spatial  frequency 
techniques  are  in  common  use.  The  IR  components  of  such  systems 
are  plagued  by  a  number  of  atmospheric  effects  that  cause 
degradation  in  the  sensed  images  driving  the  automatic  detection 
and  recognition  algorithms. 

Researchers  at  NMSU  and  the  Battlefield  Effects  Directorate  of 
the  Army  Research  Laboratory  (formerly  the  Atmospheric  Sciences 
Laboratory)  have  been  investigating  and  characterizing  these 
effects  throughout  most  of  this  research  project  [13,14].  The 
scene  degrading  atmospheric  effects  fall  into  three  broad 
categories:  1)  reduction  in  object  contrast  with  respect  to 
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background  clutter  caused  by  changes  in  energy  emissions  from 
either  the  object,  the  background,  or  both.  2)  reduction  in  object 
contrast  with  respect  to  background  clutter  caused  by  absorptive, 
radiative  and  scattering  effects  along  the  path  of  propagation.  3) 
blurring  and  distortion  of  object  details  caused  by  scattering  and 
inhomogeneities  in  the  refractive  index  along  the  path  of 
propagation. 

The  cueing  task  involves  imaging  a  relatively  large  field  of 
view  (FOV)  to  find  regions  that  have  a  high  probability  of 
containing  an  object  of  interest.  For  any  imaging  system  with  a 
fixed  number  of  resolution  elements,  FOV  and  spatial  resolution  are 
inversely  related.  A  large  FOV  implies  low  spatial  resolution  and 
vice  versa.  The  cueing  algorithm  directs  the  attention  of  the 
segmenting  and  recognizing  tasks,  allowing  them  to  operate  on 
images  with  much  smaller  FOVs  and  consequently  much  higher  spatial 
resolutions  with  respect  to  an  object's  fine  details. 

Because  of  the  relatively  lower  spatial  resolutions,  ATR 
cueing  algorithms  rely  mostly  on  contrast  measures  and  a  priori 
knowledge  of  typical  object  sizes.  Very  little  textural  and 
structural  information  is  available.  The  segmenting  and 
recognizing  algorit  ims,  on  the  other  hand,  rely  mainly  on  textural 
and  structural  properties  of  an  object  versus  background. 
Degradation  of  contrast  tends  to  have  a  large  effect  on  cueing, 
while  blurring  of  edges  and  structural  details  tends  to  affect 
segmenting  and  recognizing.  As  atmospheric  phenomena  can  affect 
both  contrast  and  definition,  it  is  important  to  characterize  these 
as  much  as  possible  to  aid  in  predicting  ATR  performance  under 
varying  conditions. 

Throughout  this  project,  data  useful  for  detailed 
characterization  of  the  degradation  phenomena  in  each  of  the  three 
categories  was  collected  and  analyzed  using  the  U.S.  Army  Research 
Laboratory's  Mobile  Imaging  Spectroscopy  Laboratory  (MISL)  with  its 
Target  Contrast  Characterizer  (TCC)[4].  The  MISL/TCC  consists  of: 
1)  two  Inframetrics  model  610,  moderately  high  resolution,  IR 
imagers;  2)  two  Sony  model  DXC-102  visible  wavelength  color  CCD 
television  cameras;  3)  a  Recognition  Concepts,  Inc.  real  time  image 
processing  system;  4)  a  computer  controlled  meteorological  station; 
5)  a  Lockheed  Model  IV-L  atmospheric  scintillometer;  6)  a  large 
area  blackbody  source;  7)  video  recording  equipment;  8) 
miscellaneous  lenses,  telescopes,  and  other  optical  components.  A 
typical  system  set-up  is  shown  in  Fig.  2. 

The  IR  imagers  are  cryogenically  cooled,  scanning  single 
detector  units  with  independent  electronic  control  over  the 
horizontal  and  vertical  scan  deflection  factors.  Each  imager  has 
two  channels  operating  simultaneously  at  wavelengths  of  3-5/im  and 
8-12m®.  The  meteorological  station  logs  temperature,  barometric 
pressure,  relative  humidity,  wind  velocity,  and  solar  flux  at  one 
second  intervals.  A  large,  highly  uniform,  blackbody  source  with 
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removable  spatial  patterns  is  used  in  a  number  of  the  experiments. 
The  unique  device  has  a  1.78m  x  1.78m  effective  surface  area  with 
a  novel  tilt-back  design  and  non-uniformly  spaced  heater 
configuration  that  eliminates  the  vertical  temperature  gradients 
and  convection  cell  separations  typical  of  large  vertical  sources. 
The  design  includes  shielding  from  solar  loading  and  wind.  The 
separation  of  the  pattern  panels  from  the  source  gives  nearly 
perfect  transitions  between  the  hot  and  cold  portions  of  the 
patterns . 

Many  of  the  theoretical  techniques  developed  in  other  areas  of 
the  ARO  Electronic  Vision  Research  project  were  applied  to  the 
actual  field  collected  data  to  assess  their  practical  utility  with 
respect  to  the  U.S.  Army's  "real  world"  problems.  Several  of  the 
concepts  proved  to  be  extremely  useful.  Complexity  and  k- 
complexity,  in  particular  are  efficient  and  robust  measures  for 
quantifying  contrast  degradations  due  to  both  inherent  scene  and 
path  environmental  effects. 

2.2  Electro-Optical  Processing 

The  goals  of  the  electro-optics  group  were  (l)  to  assess  the 
state-of-the-art  of  optical  processing  technology,  (2)  to  develop 
a  mathematical  foundation  for  designing  practical  optical 
processors,  and  (3)  to  advance  the  state  of  technology  by 
developing  and  integrating  optical  processors  with  digital  systems 
to  improve  system  performance  in  the  areas  of  monitoring,  cueing, 
segmenting,  recognizing  and/or  tracking  targets  of  interest. 

The  significant  contributions  of  the  electro-optical 
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processing  research  are  summarized  by  listing  the  most  significant 
accomplishments . 

*  Established  long-term  associations  with  the  leading  U.S. 
researchers  in  optical  image  processing  and  pattern 
recognition. 

*  Produced  the  world's  first  (a)  real  time  magneto-optic  spatial 
light  modulator  (MOSLM)  software  driver,  (b)  real  time  hybrid 
optical/digital  correlator  using  MOSLMs  as  input  and  filter 
transducers,  (c)  ternary  phase-amplitude  filters  written  on 
the  MOSLM. 

*  Developed  and  tested  a  short  coherent  optical  correlator 
architecture  using  pixelated  spatial  light  modulators  and 
identified  the  practical  limitations  of  optical  correlation 
system  miniaturization. 

*  Adapted  the  correlator  design  so  that  inexpensive  liquid 
crystal  television  displays  (LCTVs)  can  be  used  as  input  and 
filter  transducers  capable  of  continuous  modulation  of  either 
amplitude  or  phase. 

*  Produced  ternary  phase-amplitude  synthetic  discriminant 
function  composite  matched  filters  that  are  written  at  high 
speed  (up  to  300  frames  per  second)  onto  a  MOSLM  filter 
transducer  to  achieve  scale,  rotation,  and  aspect  invariant 
recognition. 

*  Developed  a  correlation  plane  filter  which  performs  a  Bayes 
likelihood  ratio  test  on  the  correlation  responses  from 
composite  matched  filters  arranged  in  a  N-dimensional  decision 
tree  to  perform  efficient  and  reliable  object  recognition. 

*  Developed,  built  and  demonstrated  a  prototype  optical/digital 
recognition  system  that  can  recognize  segmented  images  of  any 
one  of  nine  objects  placed  at  any  position  and  orientation  on 
the  NMSU  terrain  board  within  one  second  with  a  measured 
correct  recognition  rate  of  81.16%  and  a  false  alarm  rate  of 
2.4%. 

*  Demonstrated  a  real  time  LCTV-based  optical  cuer  that  locates 
targets  of  interest  in  a  large  f ield-of-view  gray  scale  image. 

*  Identified  several  future  applications  of  optical  image 
processors  in  areas  such  as  biomedical  imaging,  forensics, 
security,  air  traffic  control,  and  defense. 

2.2.1  Assessment  of  Electro-Optical  Processing  Technology 

During  the  first  year  our  efforts  were  focused  on  establishing 
the  frontiers  in  optical  processing  technology.  Several  visits 
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were  made  to  several  leading  university  and  government  laboratories 
with  active  research  programs  in  optical  processing,  intensive 
literature  searches  were  completed,  and  several  conferences  on  the 
subject  were  attended.  Perhaps,  the  most  valuable  accomplishment 
of  that  first  year  was  the  establishment  of  long-term  associations 
with  leading  research  personnel  working  on  electro-optical 
processing  projects  at  key  laboratories  throughout  the  United 
States.  A  second  major  accomplishment  was  the  interfacing  of  a 
digital  frame  grabber  to  a  magneto-optic  spatial  light  modulator 
(MOSLM)  and  the  development  of  the  world's  first  real  time  software 
package  that  allowed  images  from  a  TV  camera  to  be  digitized, 
binarized,  and  then  displayed  on  the  MOSLM  in  real  time  (30  frames 
per  second)  [15]. 

2.2.2  Hybrid  Optical /Digital  Real  Time  Correlator 

Having  established  valuable  continuing  technical  interactions 
with  government  laboratories  and  developed  a  powerful  tool  for  real 
time  optical  processor  development  (real  time  software  to  drive  the 
MOSLM)  during  the  first  year  (1987),  the  group's  attention  shifted 
to  the  development  of  a  real  time  hybrid  optical/digital  correlator 
during  the  second  year.  The  mathematical  foundation  for  such  a 
system  was  developed,  the  system  vas  built  [16]  using  128x128 
MOSLMs  at  the  input  and  filter  plants,  and  the  efficiency  of  the 
system  was  optimized.  This  system  was  the  world's  first  real  time 
128x128  MOSLM-based  binary  phase-only  sequential  correlation 
system.  One  other  researcher  (Flannery  [17]  at  the  University  of 
Dayton)  had  developed  a  similar  correlator  that  used  48x48  MOSLMs 
at  the  input  and  filter  planes  to  produce  static  (every  few 
seconds)  binary  phase-only  image  correlations,  and  three  other 
researchers  (Ross  [18]  at  Litton  Data  Systems,  Davis  [19]  at  San 
Diego  State  University  and  Psaltis  [20]  at  CalTech)  had  used  the 
MOSLM  as  a  binary  phase-only  filter  with  film  at  the  input  plane, 
but  the  NMSU  system  was  the  first  to  incorporate  128x128  MOSLMs  at 
both  planes  and  to  operate  in  real  time.  Ross  and  associates  at 
Litton,  working  independently  and  without  knowledge  of  our  work 
(and  vice  versa) ,  built  a  similar  real  time  system  that  same  year 
(1988).  NMSU  and  Litton  began  technical  interactions  during  1989. 

Three  other  major  accomplishments  highlighted  the  work  of 
1988:  (1)  the  MOSLM  frame  rate  was  increased  to  80  frames  per 
second,  (2)  time-sequenced  filtering  and  statistical  correlation 
plane  processing  techniques  were  developed  to  process  sequential 
correlation  plane  data  and  recognize  multiple  input  objects  at  any 
in-plane  rotation  [15],  and  (3)  the  world's  first  ternary  phase- 
amplitude  filters  [21]  were  written  on  the  MOSLM  and  demonstrated 
in  NMSU's  electro-optics  research  laboratory.  These  filters 
significantly  improved  the  signal  to  noise  ratio  of  output 
correlation  responses. 

During  1989  the  research  focussed  on  the  following  goals:  (1) 
increasing  the  speed  of  operation  of  the  MOSLMs,  (2)  achieving 
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invariance  to  any  number  of  image  distortions  including  aspect, 
scale,  and  in-plane  rotation,  and  (3)  reducing  the  physical  size  of 
the  optical  correlator.  The  MOSLM  frame  rate  was  increased  to  300 
frames  per  second,  and  a  rotation,  aspect,  and  scale-invariant 
object  recognition  system  was  demonstrated  using  time-sequenced 
filtering  and  statistical  correlation  plane  processing  [22,23].  A 
time-integrating  rotation-invariant  image  correlator  was  also 
developed  that  integrated  the  correlation  responses  of  sequential 
simple  filters  [24].  This  system  took  advantage  of  the  high  frame 
rate  of  the  filter  MOSLM  to  obtain  integrated  composite  responses 
that  were  then  analyzed  statistically  to  recognize  objects  of 
interest.  The  third  goal  was  realized  with  the  development  and 
demonstration  of  two  miniaturized  optical  correlators — one  using 
128x128  MOSLMs  and  the  other  using  48x48  MOSLMs  at  the  input  and 
filter  planes  [25]. 

2.2.3  Hybrid  Qptical/Diaital  Target  Recognizer 

In  1990  the  NMSU  terrain  board  was  built,  and  an  optical 
processor  was  developed  to  perform  the  target  recognition  task, 
while  digital  processors  were  developed  to  perform  the  target 
cueing  and  segmentation  tasks.  The  goals  for  1990  were  (1)  to 
build  and  characterize  two  time-integrating  correlators  using  high¬ 
speed  output  cameras  to  realize  the  advantage  of  such  systems — one 
used  a  Xybion  CID  output  camera  and  the  other  used  a  Reticon  CCD 
output  camera  [26,27],  (2)  to  simulate  the  use  of  Texas 
Instruments'  deformable  mirror  devices  (DMDs)  in  the  optical 
correlator  [28],  (3)  to  complete  the  study  of  short  optical 
correlators,  (4)  to  develop  software  to  create  composite  binary 
phase-only  filters  that  are  invariant  to  all  distortions  of  objects 
placed  at  any  position  on  the  NMSU  terrain  board  [29],  and  (5)  to 
develop  an  efficient  search  strategy  using  composite  binary  phace- 
only  filters  to  achieve  distortion  invaiiant  recognition  of  objects 
at  any  position  on  the  terrain  board.  All  of  these  goals  were 
accomplished,  and  the  optical  recognition  system  was  used  to 
recognize  selected  segmented  objects  imaged  from  arbitrary 
positions  and  orientations  on  the  terrain  board.  The  search 
strategy  selected  was  a  binary  tree  search  of  composite  binary 
phase-only  filters  [29]. 

The  goals  for  1991  were  (1)  to  demonstrate  a  working 
recognition  system  in  an  automatic  mode  using  the  terrain  board  as 
the  input  scene,  (2)  to  complete  the  study  of  the  practical 
limitations  of  optical  correlation  system  miniaturization  and  to 
build  and  test  several  systems,  (3)  to  obtain  binary  phase-only 
correlations  of  input  gray  level  images  using  a  liquid  crystal 
television  (LCTV)  as  the  input  gray  level  image  transducer,  (4)  to 
develop  an  optical  cuer  using  LCTVs  at  both  input  and  filter  planes 
of  the  optical  processor,  and  (5)  to  identify  new  applications  for 
the  optical  processing  technology  developed  at  NMSU  and  to  perform 
preliminary  experiments  in  these  application  areas. 
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The  first  goal  was  met  by  developing  a  statistical  recognition 
algorithm  based  on  a  Bayes  likelihood  ratio  test  [30,31].  The  test 
was  applied  at  each  node  of  an  N-dimensional  tree  of  composite 
binary  phase-only  filters  in  order  to  facilitate  the  correct 
traversal  of  the  tree,  correct  recognition  of  known  objects  and 
correct  rejection  of  unknown  input  objects.  An  empirical  synthetic 
discriminant  function  method  [32]  was  developed  which  produced 
composite  filters  with  nearly  equal  correlation  peak  responses  over 
the  desired  span  of  distortions.  These  filters  were  used  at  each 
node  of  the  tree  with  filters  covering  the  largest  distortion 
invariant  span  at  the  top  and  becoming  less  composite  and  more 
specific  as  the  tree  was  traversed  from  top  to  bottom.  Several 
short  optical  correlators  were  designed,  built  and  evaluated  [33] 
in  order  to  complete  the  second  goal.  The  third  and  fourth  goals 
were  accomplished  by  designing  and  building  an  optical  spatial 
filtering  system  with  LCTVs  at  both  planes  and  then  developing  and 
testing  matched  binary  phase-only  filters  for  correlation  and  size- 
selective  gray  scale  amplitude  filters  for  cueing  [34].  New 
applications  that  were  identified  and  pursued  include  correlation 
tracking  of  tissue  motion  and  blood  flow  in  ultrasound  images  [35] 
(biomedical)  and  fingerprint  classification  (forensic) . 

2.2.4 _ Bayesian  Likelihood  Classifier 

During  the  final  year  (1992)  the  electro-optical  processing 
group  concentrated  on  (1)  the  completion  and  final  testing  of  the 
Bayesian  likelihood  classifier  used  for  object  recognition,  (2)  the 
use  of  LCTVs  in  correlation  and  spatial  filtering  systems,  and  (3) 
improved  preprocessing  of  input  gray  level  images  using  spatial 
filtering  to  locate  objects  of  interest. 

After  detailed  analysis  and  testing  of  several  statistical 
classifier  algorithms,  the  composite  Bayesian  likelihood  classifier 
was  shown  to  give  the  best  performance  [36,37].  This  classifier 
exploits  the  gaussian  behavior  of  correlation  plane  response  data 
and  is  able  to  concisely  represent  multimodal  distributions 
(obtained  by  superposing  many  normal  distributions  from  thousands 
of  training  set  responses  used  as  calibration  data)  as  composite 
algebraic  functions  and  then  use  those  distributions,  to  partition 
the  vector  signal  space  into  classification  regions  derived  from 
Bayes'  likelihood  ratio  test.  The  expected  performance  of  the 
classifier  was  calculated  and  then  compared  with  the  actual 
performance  obtained  using  as  inputs  actual  training  images  chosen 
at  random.  The  predicted  overall  correct  classification  rate  was 
99.76%,  and  the  measured  rate  for  36,000  tests  of  randomly  chosen 
training  image  inputs  was  99.21%.  The  overall  correct 
classification  rate  of  terrain  board  images  (not  training  images) 
was  obtained  using  233  images  of  a  9  member  target  class  and  84 
images  of  5  nontargets  with  shapes  resembling  some  of  the  target 
class  members  randomly  placed  at  locations  covering  the  full  extent 
of  the  terrain  board.  The  terrain  board  test  yielded  an  overall 
hit  rate  of  81.16%  and  a  2.4%  false  alarm  rate. 
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LCTVs  were  characterized  and  used  to  produce  several  types  of 
filters  including  matched  binary  phase-only  filters,  gray  scale 
apodized  amplitude  filters,  and  amplitude  weighted  binary  phase- 
only  filters.  LCTVs  were  also  used  to  produce  gray  scale  input 
images  to  test  each  of  the  filtering  operations.  A  real  time 
target  cuer  was  built  [38]  and  used  successfully  to  locate  targets 
of  interest  in  a  large  f ield-of-view  gray  scale  image  using  size- 
selective  spatial  filters  with  Hanning  apodization  to  reduce  the 
ringing  in  the  filtered  output  images.  The  inputs  and  filters  were 
written  to  LCTVs,  and  the  output  bright  spots  corresponding  to  the 
detected  targets  of  interest  were  acquired  with  a  CCD  camera. 

2.3  Multiprocessor  Vision  Architecture 

The  goal  of  the  multiprocessor  architecture  research  is  the 
development  of  a  methodology  for  designing  and  evaluating 
multiprocessor  architectures  for  real  time  vision  systems.  The 
major  accomplishments  of  this  research  effort  are: 

*  Discovery  of  a  virtually  unexplored  class  of  multiprocessor 
architectures  whose  members  exhibit  superior  performance  for 
many  real  time  weapon  systems  and  battlefield  management 
tasks,  including  multisensor  vision.  This  class  of 
multiprocessors  employ  global  memory  for  high  speed 
computations,  and  message  passing  for  modularity,  reliability, 
and  scalability  [11]. 

*  Design  and  construction  of  a  prototype  Virtual  Port  Memory 

machine  [12]  that  supports  the  global-memory  and  message¬ 
passing  architecture.  Evaluation  of  this  machine  has 

confirmed  the  performance  predicted  analytically. 

*  Development  of  the  ARGOS  multiprocessor  operating  system  for 
Global-Memory  Message-Passing  architectures. 

*  Establishment  of  the  Parallel  Architecture  Research  Laboratory 
for  studying  the  theory  and  design  of  multiprocessor  and 
distributed  architectures. 

*  Development  of  a  theoretical  model  of  parallel  systems  that 
predicts  system  performance  based  upon  nearly  independent 
models  of  architectures  and  algorithms. 


2.3.1  The  Virtual  Port  Memory  Machine 

Our  contributions  to  the  field  of  multiprocessor  hardware 
design  include  the  development  of  the  prototype  Virtual  Port  Memory 
machine,  and  the  evaluation  of  related  ideas  in  processor  and 
system  architecture. 

The  Virtual  Port  Memory  (VPM)  machine  was  built  specifically 
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to  evaluate  the  novel  idea  of  supporting  a  message  passing  parallel 
environment  using  a  global  memory  hardware  structure  [39].  This 
unusual  combination  provides  the  modularity  and  scalability  of 
message  passing,  while  using  a  shared  global  memory  to  provide 
uniform  high-speed  access  to  the  large  data  structures  found  in 
computer  vision.  Because  process  address  spaces  are  strictly 
private  in  the  message  passing  paradigm,  processors  are  free  to 
employ  very  large  cache  memories  to  improve  system  performance, 
without  a  requirement  to  maintain  cache  consistency  [40], 

Analysis  and  simulation  of  a  prototype  design  indicated  that 
such  a  machine  (using  CISC  processors)  would  scale  to  256 
processors  on  a  single  bus,  and  provide  higher  performance  than 
other  architectures  using  similar  technology.  To  test  these 
predictions,  we  built  a  small  prototype  machine,  and  wrote  a 
POSIX-compliant  operating  system  for  it  (ARGOS,  described  below) . 

Gaussian  elimination  was  chosen  as  an  initial  vehicle  for 
evaluating  the  performance  of  the  VPM  prototype,  due  in  part  to  the 
availability  of  published  data  for  the  performance  of  both  message 
passing  and  shared  variable  versions  of  this  algorithm  on  the  BBN 
Butterfly  multiprocessor.  A  message-passing  version  of  the 
algorithm  was  written  in  C  for  the  VPM  prototype,  and  its  execution 
on  a  single  processor  was  measured  using  the  facilities  of  the 
Parallel  Architecture  Research  Lab  (PARL)  at  NMSU.  These  detailed 
measurements  were  then  used  in  a  simulation  of  VPM  machines  with  up 
to  128  processors  (the  maximum  number  of  processors  in  the 
Butterfly  results) ,  and  various  analyses  were  performed  on  the 
simulation  results. 

These  results  confirmed  our  earlier  analytical  predictions  of 
linear  speedup  for  VPM  machines  with  over  100  processors,  even  when 
a  bus  is  used  as  the  processor-memory  interconnection  network.  The 
speedups  achieved  with  the  800  x  800  Gaussian  elimination  on  the 
VPM  machine  matched  the  theoretical  maximum  speedup  available  from 
the  algorithm,  while  the  Butterfly  speedups  fell  far  short  of  those 
theoretically  achievable  due  to  memory  and  interconnection  network 
contention  [48]. 

In  addition  to  the  advantages  of  the  Virtual  Port  Memory 
multiprocessor  architecture  for  such  computation-intensive  tasks  as 
Gaussian  elimination,  we  have  found  it  well  suited  for  the  I/O 
intensive  tasks  of  computer  vision,  as  well  as  for  such  similar  I/O 
intensive  applications  as  processing  space  telemetry  packets  (which 
are  expected  to  arrive  at  1200  Mbps  from  the  TDRS  system;  see 
below) .  Furthermore,  it  solves  some  of  the  most  vexing  problems 
involved  in  designing  a  general  purpose  multiprocessor  for 
mainstream  commercial  applications. 

Major  computer  manufacturers  are  currently  exploring 
multiprocessor  architectures  for  commercial  applications,  and  are 
finding  the  conventional  answer  of  bus-based,  cache-coherent 
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architectures  unattractive,  due  both  to  the  difficulty  of  designing 
cache  coherent  machines  using  today's  most  powerful  processors,  and 
in  the  inherent  non-scalability  of  such  designs.  These  problems 
are  exacerbated  by  the  ability  of  current  super-scalar  processors 
to  issue  multiple  instructions  per  clock  cycle,  and  thereby 
individually  consume  most  of  the  bandwidth  of  a  bus. 

Although  the  simple  bus-based  Virtual  Port  Memory  architecture 
is  unsuited  for  such  applications,  related  GMMP  (global 
memory-message  passing)  architectures  have  adequate 
processor-memory  bandwidth  and  don't  require  cache  coherency,  thus 
making  such  GMMP  architectures  quite  attractive  for  scalable 
high-performance  multiprocessors . 

IBM,  Convex  (the  leading  manufacturer  of 
"mini-supercomputers") ,  and  Solbourne  (a  leading  source  of  RISC 
multiprocessor  server  machines)  have  expressed  interest  in  GMMP 
architectures  for  future  multiprocessor  products. 

2.3.2  Related  Contributions 

Our  basic  research  into  the  uses  of  video  RAM  (VRAM)  ,  with  its 
independent  random-access  and  sequential-access  ports  [42],  found 
application  in  multiprocessors  capable  of  handling  high-rate  I/O 
[43].  An  opportunity  to  evaluate  this  technique  was  available  in 
a  NASA  program  seeking  to  identify  architectures  capable  of 
processing  TDRSS  packets  which  arrive  at  the  White  Sands  Ground 
Terminal  over  four  concurrent  300  Mbps  channels. 

We  developed  a  design  in  which  the  high-rate  data  is 
pre-processed  by  special-purpose  I/O  controllers  to  perform  frame 
synchronization,  error  correction  and  related  bit-level  functions, 
and  is  then  loaded  into  the  VRAM  main  memory  of  a  modified  VPM 
machine  through  the  VRAM  sequential  access  ports,  whose  bandwidth 
significantly  exceeds  even  the  1200  Mbps  figure.  General  purpose 
processing  elements  then  process  packet  headers  and  route  the 
packets  to  terrestrial  links  by  normal  accesses  via  the  VRAM  random 
access  port,  and  send  messages  to  outbound  I/O  controllers  which 
read  processed  packets  out  through  VRAM  sequential  ports  for 
terrestrial  transmission. 

Other  research  projects  in  multiprocessor  hardware  design 
include  a  VLSI  address  translation  cache  device  [44,45]  and  a  VLSI 
implementation  of  a  virtual  memory  page  replacement  algorithm  [46]. 
In  addition,  we  designed  and  evaluated  a  novel  CPU  architecture 
that  tags  all  instructions  and  operands  to  ensure  data  integrity 
and  correct  program  operation  [47], 

A  technique  called  "Working  Set  Prefetching"  [48]  was 
developed  that  improves  multiprocessor  performance  by  eliminating 
redundant  cache  line  fetches  over  system  buses.  (Further 
development  of  this  technique  is  the  subject  of  a  proposal  to  the 
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Strategic  Defense  Initiative  Organization.) 

2.3.3  The  ARGOS  Operating  System 

Our  experimental  GMMP  operating  system  ARGOS  (A  Research  GMMP 
Operating  System)  was  implemented  both  to  support  the  evaluation  of 
the  Virtual  Port  Memory  architecture,  and  to  explore  the 
implications  of  supporting  message  passing  (with  private  address 
spaces)  on  a  global  (shared)  memory  machine  [49,  50,  51,  52].  The 
performance  evaluation  discussed  above  used  the  actual  Local  Agent 
(kernel)  message  passing  code.  Some  interest  has  been  expressed  in 
our  operating  systems  work  by  industrial  researchers,  and  one  of 
the  students  who  implemented  ARGOS  was  hired  by  IBM’ s  workstation 
division  to  continue  to  work  on  novel  operating  systems. 

2.3.4  _ Parallel  Architecture  Research  Lab 

The  Parallel  Architecture  Research  Laboratory  (PARL)  was 
established  under  this  grant  to  provide  a  facility  for  designing, 
building,  programming,  and  evaluating  multiprocessors.  ARO  funding 
has  achieved  significant  leverage  through  subsequent  equipment 
grants  from  leading  computer  manufacturers,  including 
Hewlett-Packard,  Tektronix,  and  Advanced  Micro  Devices. 

In  PARL,  we  have  established  a  unique  facility  for  acquiring 
and  analyzing  computer  address  traces.  Simulation  using  such 
traces  is  a  standard  technique  for  evaluating  new  concepts  in 
computer  architecture  under  realistic  workloads.  Our  facility 
includes  a  dedicated  Unix  workstation  (with  a  CISC  CPU)  which  is 
instrumented  for  collecting  complete  traces  (including  operating 
system  and  interrupt  activity),  a  real  time  trace  filter  for 
compressing  traces  and  extending  the  coverage  of  finite  trace 
buffers,  a  data  acquisition  system  capable  of  storing  512K  to  1 
million  samples  per  acquisition,  and  a  wide  variety  of  trace 
analysis  and  simulation  programs.  The  prototype  Virtual  Port 
Memory  machine,  as  well  as  other  parallel  machines  on  campus,  serve 
as  sources  of  multiprocessor  traces. 

An  address  trace  compaction  module  was  built  to  expand  the 
length  of  address  traces  we  can  acquire  with  our  DAS-9200  from  128K 
references  to  1M  references,  which  crosses  the  threshhold  for 
statistical  usefulness  in  many  memory  hierarchy  simulations. 
Following  this,  a  real-time  trace  filter  was  built  [53],  which  can 
facilitate  the  collection  of  virtually  unlimited-length  traces  by 
reducing  the  average  trace  data  rate  to  within  the  bandwidth  of  a 
local  area  network,  allowing  the  use  of  mass  memory  to  store  traces 
rather  than  expensive  static  RAM  (which  is  necessarily  of  limited 
capacity) . 

2:?t5 _ Performance  Modeling  and  Analysis 

One  of  the  first  products  of  this  new  facility  is  a  new  model 
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of  the  cache  miss  inter-arrival  process.  Analysis  of  traces  from 
the  SPEC  benchmark  suite  (on  both  CISC  and  RISC  workstations) 
suggests  that  this  process  may  be  well  modeled  as  a  random  process 
with  inverse-Gaussian  inter-arrival  times  [54,55].  We  found  this 
model  to  be  significantly  more  accurate  than  the  usual  assumption 
that  cache  misses  irrive  from  a  memoryless  process  (e.g.,  an 
exponential  distribution) . 

Che-Chi  Weng  is  analyzing  traces  produced  by  this  facility  to 
develop  a  model  of  instruction  access  patterns,  which  can  be  used 
to  produce  synthetic  programs  for  exercising  a  wide  range  of 
computer  system  models.  This  workload  characterization  research  is 
expected  to  make  a  valuable  contribution  to  the  field  of 
multiprocessor  performance  modeling,  and  was  discussed 
enthusiastically  by  a  number  of  the  attendees  at  the  1991 
SIGMETRICS  conference. 

In  the  final  year  of  the  grant,  a  technique  for  predicting  the 
performance  of  parallel  systems  was  designed  that  can  combine 
performance  results  from  separately  evaluating  parallel  algorithms 
and  architectures  into  a  prediction  of  how  well  particular  codes 
will  run  on  particular  machines  [56].  (Continuation  of  this 
project  is  the  subject  of  a  pending  proposal  to  ARO.)  Ultimately, 
this  technique  could  lead  to  an  “architectural  spreadsheet*  that 
allows  a  system  integrator  to  tune  a  program  and  an  architecture  — 
with  real-time  feedback  —  to  obtain  peak  performance  before 
hardware  actually  exists. 


2jiA _ Visjpp  gygtems/ Intelligent  Control 

The  control  systems  research  focused  on  the  development  of 
adaptive  structures  and  control  concepts  for  modeling,  designing 
and  evaluating  multisensor  vision  control  systems.  The  first  part 
of  the  research  focused  on  the  development  of  system  structures  and 
models  to  fuse  multisensor  scene  information.  The  second  part  was 
concerned  with  the  development  of  intelligent  controller  structures 
for  controlling  highly  non-linear,  difficult  to  model  processes. 
The  major  accomplishments  of  these  research  efforts  are: 

*  Development  of  a  system  structure  and  model  for  fusion  of 
multisensor  scene  information  to  perform  scene  interpretation 
and  understanding  for  intelligent  control  decisions  [57,58]. 
A  layered  and  modular  hierarchical  structure  for  the 
intelligent  controller  [59]  was  developed  and  several  modules 
in  the  different  layers  of  the  control  structure  were 
developed.  The  major  modules  developed  for  the  control 
structure  are: 

/  A  blackboard  Relational  Data  Base  Management  System 
(RDBMS)  Structure  for  handling  and  organizing  a  large 
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number  of  sensor  reports  [58,59,60,61]. 

✓  Sensor  information  fusion  modules  based  on  RDBMS  concepts 
for  scene  interpretation  and  understanding  under 
uncertainty  [62,63,64]. 

/  Multispectral  fast  object  recognition  systems  for 
improved  performance  and  reliability  [58]. 

/  Sensor  performance  evaluation  and  adaptive  sensor 
selection  based  on  consensus  and  recent  performance 
evaluations  [65,66]. 

/  Data  association  for  trajectory  estimation  and  tracking 
[67,68]. 

✓  Robust  distributed  multisensor  data  fusion  that  combines 
the  strengths  and  minimizes  problems  of  serial  and 
parallel  approaches  [69]. 


*  Development  of  a  test-bed  for  designing  and  evaluating  non- 
classical  control  techniques  for  highly  non-linear,  difficult 
to  model  plants  [70].  A  vision  control  system  for  remotely 
backing  a  tractor-trailer  in  a  constrained  space  was 
implemented.  The  system  has  a  replaceable  controller  for 
evaluating  different  controller  approaches.  Three  intelligent 
control  techniques  were  developed  to  solve  the  trailer  back-up 
problem.  These  are: 

✓  K-Nearest  neighbor  controller  using  neural  networks  [10] . 

/  Rule-Based  Expert  system  controller  [70] . 

/  State  Partitioning  variable  structure  controller  [70]. 

2.4.1  Multisensor  Scene  Interpretation  and  Understanding 

Research  was  focused  on  the  development  of  a  system  structure 
and  model  to  fuse  multisensor  scene  information  for  intelligent 
control  decisions  [57],  A  layered  and  modular  distributed 
hierarchical  structure  for  the  intelligent  controller  was  developed 
[59].  The  control  structure  is  comprised  of  three  layers.  The 
primary  layer  (core)  does  the  high  level  decision  making  and 
performance  evaluation.  The  secondary  layer  performs  data  fusion, 
scene  interpretation  and  reduction  of  dimensionality.  This  layer 
also  provides  a  communications  link  between  the  primary  and 
tertiary  layers.  The  tertiary  layer  is  responsible  for  low-level 
signal  processing  (sensors  and  actuators)  and  sensor 
characterization.  These  layers  are  made  up  of  modules  that  perform 
specific  tasks  assigned  to  the  different  layers.  This  control 
structure  organizes  a  large  number  of  sensor  reports  in  a 
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blackboard  Relational  Data  Base  System  to  facilitate  management  and 
operation.  Under  this  structure,  sensor  information  fusion,  scene 
interpretation  and  understanding,  and  control  decision  were 
implemented  as  relational  operators  operating  on  the  appropriate 
relations  (tables) .  A  set  of  operators  in  the  lower  layers 
operates  on  relations  formed  from  the  incoming  sensor  reports  to 
produce  new  relations  to  be  operated  on  by  the  next  layer  of 
operators.  This  process  is  repeated  in  each  layer  until  an 
intelligent  control  decision  is  reached.  The  resulting  structure 
provides  a  flexible  means  for  incorporating  redundancy  and  allows 
different  schemes  to  accomplish  the  same  task  thus  increasing 
system  reliability. 

A  set  of  operators  were  developed  to  perform  several  tasks  at 
different  levels  of  the  hierarchy.  In  the  area  of  object 
recognition  for  scene  interpretation,  two  2-D  object  recognition 
systems  were  developed  [60,61].  A  robust  multispectral  global 
object  recognition  system  [58]  and  a  fast  partially  occluded  object 
recognition  system  [71,76].  Both  of  these  systems  were  based  on 
Normalized  Interval  Vertex  Descriptors  [72].  In  the  area  of  scene 
interpretation,  a  set  of  relational  operators  for  fusing  numerical 
and  symbolic  information  under  uncertainty  was  developed 
[62,63,64,73].  These  operators  are  based  on  concepts  from  the 
Dempster-Shafer 1 s  Theory  of  Evidence  [11].  Using  "confidence 
factors",  sensor  consensus,  and  sensor  performance  history,  an 
adaptive  sensor  selection  and  sensor  performance  evaluation  system 
was  developed  [65,66].  A  new  technique  for  distributed  multisensor 
data  fusion  was  also  conceived  [69],  This  idea  combines  the 
strengths  and  minimizes  problems  associated  with  serial  and 
parallel  approaches  for  distributed  data  fusion.  In  the  area  of 
data  association  and  tracking,  several  modules  were  developed. 
Starting  from  the  standard  Kalman  filter  techniques  for  reference, 
to  faster  and  more  sophisticated  non-statistical  techniques 
[67,68]. 

In  an  effort  to  tie  research  findings  to  a  more  realistic 
implementation  of  a  multisensor  control  system,  a  computer 
simulation  model  of  the  Transportable  Telemetry  Acquisition  System 
(TTAS)  of  the  WSMR  Multisensor  Tracking  System  was  also  developed 
[74]. 

2.4.2  Intelligent  Controller  Structures 

The  second  part  of  the  control  system  research  was  concerned 
with  the  development  of  intelligent  controller  structures  for 
controlling  highly  non-linear,  difficult  to  model  processes  [70]. 
Designing  controllers  for  highly  non-linear  processes  is  among  the 
most  challanging  and  difficult  of  control  problems.  Starting  from 
the  fact  that  a  "good"  model  is,  in  most  cases,  difficult  to 
obtain.  Even  if  such  a  model  is  obtained,  designing  and 
implementing  an  actual  controller  based  on  the  model  remains  an 
even  more  significant  problem.  Most  multisensor  control  systems 
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fall  into  this  category.  Most  of  these  types  of  controllers  are 
also  goal  oriented,  making  the  development  of  an  intelligent 
controller  a  natural  step  for  handling  these  types  of  situations. 
One  of  the  main  drawbacks  of  intelligent  controllers  is  their  lack 
of  generality,  that  is,  they  are  in  most  cases  application 
dependent.  For  this  reason,  we  defined  a  problem  that  encompasses 
most  of  the  previously  mentioned  constraints.  Due  to  its  relevance 
in  the  current  literature,  the  problem  selected  was  the  evaluation 
of  the  controllers  used  for  backing  a  tractor-trailer  in  a 
constrained  space  [75].  The  goal  of  each  controller  evaluated  is 
to  back  a  tractor-trailer  to  a  loading  dock  independently  of  the 
trailer  load  and/or  the  parking  lot  conditions.  Goal  orientation 
and  model  independence  are  the  driving  forces  behind  this  research 
effort.  With  these  ideas  in  mind,  we  developed  a  suite  of 
intelligent  controllers  to  carry  out  the  task.  They  included:  a  k- 
nearest  neighbor  network  controller  [10],  a  rule-based  expert 
system  controller  [70],  and  a  state  partitioning  variable  structure 
controller  [70]. 

The  k-nearest  neighbor  controller  was  implemented  using  a 
"neural-like"  hierarchical  structure.  The  overall  goal  of  the 
system  (backing  to  the  dock)  was  subdivided  into  simpler  goals  with 
associated  secondary  controllers.  The  goal  of  an  individual 
secondary  controller  was  to  take  the  system  closer  to  the  overall 
goal.  The  controllers  were  trained  using  data  derived  from  a 
simplified  computational  model  of  18-wheel  tractor-trailer  rig.  In 
the  initial  two  level  implementation,  the  top  level,  called  the 
supervisor,  assesses  the  control  situation  and  selects  the 
appropriate  controller  in  the  second  level  to  control  the  system. 
The  controllers  in  the  second  level  are  very  simple  k-nearest- 
neighbor  (KNN)  controllers  that  perform  relatively  simple  tasks. 
To  demonstrate  the  concepts,  an  "intelligent"  vision  guided  system 
was  developed  to  remotely  park  a  tractor-trailer  in  a  constrained 
region.  This  controller  requires  only  eight  control  points  and  a 
rule-based  expert  system  supervisor.  The  control  structure  adapts 
well  to  random  effects  caused  by  measurement  and  vehicle  control 
errors  and  is  suited  for  nonlinear  and  unstructured  control 
problems  that  are  difficult  to  model.  One  of  the  major 
accomplishments  of  this  approach  is  the  reduction  of  the  complexity 
of  the  lower  level  controllers,  since  they  are  designed  to  control 
the  system  in  very  simple  operating  modes.  For  this  particular 
implementation  the  secondary  controllers  consisted  of  only  nearest 
neighbor  control  elements.  This  controller  was  also  successfully 
used  to  balance  a  inverted  pendulum. 

A  Rule-based  expert  system  controller  was  also  implemented  to 
for  the  trailer  backing  problem.  Knowledge  gained  from  an 
experienced  tractor-trailer  driver  was  converted  to  IF-THEN  rules 
to  back  the  truck  to  the  dock.  These  rules  were  stored  in  a  rule 
database  and  later  used  to  guide  the  truck  under  the  expert  system 
control. 
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A  general  approach  was  developed  for  this  type  of  control 
problem.  The  approach  termed  "state  partitioning  control"  involved 
the  idea  of  a  multiple-level  variable  structure  controller  where 
the  control  problem  (control  surface)  is  partitioned  into  simpler 
control  problems  that  can  be  solved  with  simpler  controllers.  The 
top  level,  the  supervisor,  assesses  the  control  situation  selecting 
the  appropriate  controller  in  the  secondary  levels  to  control  the 
system.  The  flexibility  of  this  approach  is  that  any  controller, 
either  the  supervisor  or  any  of  the  secondary  controllers,  can  be 
implemented  in  a  different  manner  depending  on  the  control 
situation  (rule-based,  neural  networks,  classical  controller,  non¬ 
linear  controller,  etc.).  For  this  particular  problem  a  two-level 
controller  was  implemented.  The  supervisor  and  the  secondary 
controllers  were  implemented  using  rule-based  expert  systems.  The 
big  difference  between  this  implementation  and  the  previous  global 
expert  system  lies  in  the  number  and  complexity  of  the  rules  to 
control  the  truck.  In  the  global  case  a  larger  number  of  more 
complicated  rules  are  required  since  there  are  more  complex  control 
situations.  In  the  state  partitioning  case,  the  control  situations 
are  much  simpler  (forward,  backwards,  etc.)  therefore  a  smaller 
number  of  simple  rules  are  required. 

Another  very  important  drawback  of  intelligent  controllers  is 
the  lack  of  mathematical  tools  for  performance  analysis  and 
comparisons.  To  overcome  these  difficulties,  we  developed  a  test¬ 
bed  for  implementation  and  development  of  non-classical  control 
techniques.  A  vision  control  system  to  control  the  backing  of  a 
"pseudo-real"  truck  was  implemented.  This  system  was  designed  with 
a  modular  replaceable  controller.  The  system  has  the  capability  of 
controlling  the  plant  with  different  controller  schemes  for 
performance  comparisons.  The  experiments  consisted  of  backing  a 
remotely  operated  toy  truck  and  trailer  controlled  by  a  personal 
computer  under  the  guidance  of  a  vision  system  using  the  various 
controller  modules. 
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An  important  goal  of  the  research  project  is  to  transfer  new 
concepts  and  techniques  to  the  Army  laboratories  and  industry.  Often 
this  requires  the  investigators  to  assist  the  laboratories  implement 
the  concepts  in  their  applications.  The  feedback,  however,  is 
important  in  evaluating  and  motivating  the  basic  research.  Some 
important  technology  transfer  activities  related  to  the  sponsored 
research  are: 

*  Several  members  of  the  image  processing  and  electro-optical 
research  groups  have  worked  with  Duke  University  Engineering 
Research  Center  for  Emerging  Cardiovascular  Technologies  on 
several  joint  projects  for  applying  digital  and  optical  signal 
processing  techniques  to  cardiovascular  problems.  The  effort  is 
a  collaborative  real-time  signal  processing  research  in 
automatic  implantable  defibrillators  and  ultrasonic  blood  flow 
measurements . 

*  The  image  processing  research  group  assisted  the  Atmospheric 
Science  Laboratory's  Target  Contrast  Character izer  project  team 
in  implementing  a  near  real-time  complexity  measuring  system  for 
IR  signature  analysis.  The  system  is  being  used  to  investigate 
and  characterize  atmospheric  and  battlefield  conditions  as  they 
affect  IR  target  detection  equipment. 

*  The  image  processing  research  group  is  assisting  the 
Instrumentation  Division  at  White  Sands  Missile  Range  design  a 
near  real-time  film  reading  processing  system  using  the  results 
of  the  research  sponsored  by  this  research  contract. 

*  The  image  processing  research  group  together  with  the  NMSU 
entomology  department  are  working  to  develop  an  electronic 
vision  system  for  automatic  assessment  of  cotton  insect 
pest  densities. 

*  The  electro-optical  research  group  is  applying  the  N-dimenr.ional 
tree  search  methodology  to  the  output  response  data  from  the 
acousto-optic  image  correlator  at  Sandia  National  Laboratories. 

*  The  electro-optical  research  group  demonstrated  the  feasibility 
of  using  both  joint  transform  correlators  and  acousto-optic 
correlators  to  measure  flow  vectors  in  medical  ultrasound  images 
obtained  from  Duke  University. 

*  The  architecture  research  group  assisted  several  government  and 
industrial  organization  evaluate  the  Virtual  Port  Memory 
Multiprocessor  Architecture  relative  to  their  applications. 
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1  Technical 

contacts  made  during  the  grant  are: 

|  NAME 

ORGANIZATION 

APPLICATION 

■Tom  Reader 

U.S.  Army  Vulnerability 

Multispectra 1  Evaluation 

■ 

Assessment  Laboratory 

■Robert  Green 

WSMR  Instrumentation  Div. 

Target  Tracking 

■Henry  Stephenson 

and  Fusion 

■Terry  Stalker 

Sandia  National  Labs 

Autonomous  Vehicles 

■Fred  Dickey 

Albuquerque 

Jeffrey  Carlson 

Sandia  National  Labs 

Tracking  Applications 

■Don  Gregory 

MICOM,  Huntsville 

Missile  Systems 

■Jim  Florence 

Texas  Instruments 

Phase  Modulation 

1 

Phillippe  Backer 

IBM  Advanced  Workstations 

Operating  Systems 

|james  Dillinger 

IBM  Advanced  Workstations 

Virtual  Port  Arch. 

James  Kahle 

■ 

IBM  Advanced  Workstations 

Virtual  Port  Arch. 

■Lynn  West 

IBM  Advanced  Workstations 

Virtual  Port  Arch. 

^Robert  Gallegos 

Bell  Laboratories 

Multiprocessor  OS 

Karl  Whiting 

Convex  Computer  Co. 

Virtual  Port  Arch. 

^Philip  Fleming 

AT&T 

Performance  Analysis 

Blaine  Gaither 

■ 

Amdahl  Corp. 

Performance  Analysis 

■Rajeev  Jog 

Hewlett-Packard 

Performance  Analysis 

jDan  Welter 

Compaq  Computer  Corp. 

Performance  Analysis 

Adrian  King 

Sandia  National  Labs 

Parallel  Computing 

^Stephen  Cook 

Australia 

Automated  Communication 

Robert  Desourdis 

■ 

Science  Applications 

Automated  Communication 

■Simon  Rosenblatt 

Army  Info  Systems  Eng. 

Automated  Communication 

■Lars  Stromberg 

Sunair  Electronics 

Automated  Communication 

Wendell  Watkins 

Atmospheric  Sciences  Lab. 

IR  Image  Complexity 

1 

1 
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Automatic  Film  Reader 


■ion  Hayslett 
Joe  Ambrose 

^Bill  Ross 

Jsreg  Trahey 

Tom  Cathey 
■lark  Freeman 


WSMR  Instrumentation  Div. 

Litton  Data  Systems  Phase-Only  Filtering 

Van  Nuys ,  CA 

Duke  University  Biomedical  Research 

University  of  Colorado  Optical  Computing  Center 


